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2404054 

RACK-LEVEL POWER MANAGEMENT OF COMPUTER 

SYSTEMS 

Sachin NairtnChhada: Loren tJL Koohto; Robert Wffllam Dobbs 



BACKGROUND OF THg JMVgWTinM 
10 Reldofthufnw^H^ , 

ITie praaent invention relates generally to power supplies for 
eledRNdcs and computerB. 

19 Description of ttiG Bacfcmpmvi A4 

CompiA^r nbtwDrics. such as Ibe Internet, utilize hlQh perfoimance 
computer systems called 'sewers.' Servers typically have hlgti perfbnnanca 
proceeeors and contain hardware and software capable of handSng lai^ 
2D amowite of data. Servers provide a large number of computer usere with 
compute services and access to large stores of Infonrnatkm. For example, 

servere are used to host web sMee that can be accessed by many computeis via 
thebitemet 

Miltiple senrar components ere often housed wHbln a setver 
25 hotisfttg or "^nrer Fads'. Server radcs are typically box-«lce structures 

cabinets Ihet contain a number of rsmoyable eteetrenlc modules or etoctronic 

trays. Each electrxniic tray can be a dlfferaiit server, or each electronic tray can 
hold one or more components of a server. 

Other types of computing systems include bleded computers, 
30 woflcstations. and other oomputino machines. Those computing systems may be 
in various arrartgemonts. end are not neoessarfly m a recked configuration. 

The above discussed saner radcs, blade chassis, worfcstetlon 
groups, and eknilar systems require substantial power to operate, and 
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conttnuous operation of the computing systems Is of critical importance in data 
centars. office envtronments, and other applications. Hence. Improvements In 
systems and methods for providing power to systems wHh multiple computsis 
arehlohiydeslnable. 

suMi\iARY 

One embodiment of the invention pertains to a system tbr power 
management of a giDUp of computeis. The ^stemindudes server side 
infrastnjdure (SSI) cBcuHiy at each computer m the group and a centranzad 
power management module (CPMIW). The SSI droiiiiy Includes local monitoring 
drcolny coupted to a central processing unH (CPU) of the oomputer. The CPMM 
has a manaoement link to the SSI circuHjy at each computer in the group. The 
local drcuKiy at each computer monKors power consumption at the CPU of that 
computer and transmits power, consumption data to the CPMM. The CPMM 
16 applies a sat of niles to the power consumption data to delemiine when and at 
wWch computers to enable and disable a CPU power throttling mode. 

Another embodiment of the hiventjoh pertains to a sewer-sidB 
apparatus ft>r a ladMnountad computer. Local monKodngclicuitiy at the 
computer IS ooupled to a central processing unit (CPU) of the computer and 
coupled to a centraDasdpowermanagement system. Thetocalotoiitiyis 
configumd to monitor powo- consump&m at the CPU, tiansmit power 
consumption data to the centnalized power managbment system, receive 
manasoment messages from the centralized power management system, and 
send eommarids to enable and disable a power throttUng mode at the CPU. 

Another embodiment of the Invention peitains to a central power 
management appar^ for a group of computers mounted m a rack. A 
nrtanagement module is coupled to local monitoring drcuitiy at each computer fai 
thegroup. The management module is conjured to receive power 
consumption data ftom the local monitoifngdrcuiby, determine at which 
compuiere to enable and disable a CPU powerthroUing mode, and transmit 
massages to said detemnined computers to enable and disable the CPU power 
throtti«tig mode. 

Another embodiment of the invention pertains to a method for 
power management of a group of computefs. Power corsunption at each 

2 



20 



2ff 



PAGE7/16'RCVDAT2l2SI20(l$2:S8:36PM[EastemStandardTiim]'SVI^ — 



FEB-28-2006 13:01 ' HP LEGRL FT COLLINS 970S987247 P. 08 



computBT In ths group is monllored. The twullant power'consumpiiein (fata is 
tiansmitted from each computer In the group to a centralized power manager. 

Another embodiment of the invention peitalns to a centralized 
method fw managing power consumption of 8 group of computera. Power 
* consumption data is received from the local roonitorino drouitry. A detennlnation 
Is mada as lo at which computers to enable and dteabte a ciPU power-OiroltHno 
mode. Messages are tranemhted to said detenn'med computere to enable and 
disable the CPU power thfoitlb« mode. 

Another embodtrnenl ofthe InvenUon pertains to a power 
io managemenlapparatusformanaglngpowerusaoeofofliwpofcomputersaia 
rack-JevBl. The appannuslndudee at least meana for ieeeh*ig power 
consumption data iiom the local monHortng dreuttiy, means for deiennMi^ at 
which computers to enable and disabte a CPU power throttling mode, and 
means for transmitting messages to said detsmilned computeis to enable and 
»5 disable the CPU power throttling mode. 

PRIEF DESCRIPTIQM OF THF riPAWi^^y 

RG. 1 is a schematic diagiam deplcilno sorter side infiasbuctuie 
20 cireultiylnacoafdancewiih an embodimentof the invention. 

FIG. 2 is a schematic diagram deplolhg a oenlialtced power 
management module coupled to mutUple seivers in aooordanoe with an 
embodiment crfthe inventioa 

^ PgrAlLgPr^g^gfpRfpTin^ 

As discussed above, senrar radcs and similar systems require 
stAstantlal power to opernte, and conihvjous operation of (he senwis is of 
critical importance* data centers, compute terns, and other appiieaUons. 
30 Ftocentfy. hfgh^ensity computing systems, for example implemented us&ig 
blade seiVBis, are becoming more pievalsm. Applicants have found that such 
MgbHJensity computing fscka may have power requirements beyond vvhat the 
older Inlrastnjcttire of the data oentore was designed to a ccom m odate. 
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To ovsrcome thl$ problem, a poUdng mechanism may be used to 
enable using these new Ngher-waltaga servere in existing data oentere or 
compute liarms. One embodiment ofthepresertlnvenltonUiipBts for polWna 
the parts of the system wffli the tiighest power consumption, Itose parts being 
5 the central processing units (CPUs) of the sen^eni. Another embodiment 

employs a CPMM to trade power utiKzetibn over time, and uses this bifbmiaUon 
In conjunction with other parametere (ibr example, CPU temperature) Ibr pofidi^ 
power untzalion. 

Ktonitoring and throttling power consumption at the CPU level is 
10 advantageous over previous solutions, which Involved application level 

nxmitocing and throttling. This is because application monitoring and throttling 
requires slgnilicanl changes and adjustments be made to the application and/or 
operating system code. Instead, monHoring and tfyottilng power consumed by 
CPUs Is transparent to the applicatjons lunniiig on the computer and does not 
IS requlna such code modlRcation. 

RO, 1 1s a schematte diagi^m depicting server side inlrastrudurB 
(SSI) circuitiy 100 In accordance wHh an embodiment o^ the invention. The SSI 
circuitry 100 at one server or computing device id shown. Each server or 

computing device on a racl< or blade chassis may be configured to hdode such 
20 SSI circuitiy 100. 

The SSI drcuiby 100 operates to monitor and throttle the power 
consumption ofa CPU 102 oftheseiveror computing device. One component 
of Ote SSI diculliy 100 Is the local monitoring drcuiby 104. The local monitoiing 
drcultry 104 may be Implemented using a nricrooontroller and other drcuMry. 
2S ''^ local mordtoring drcultry 104 Is coupled to the CPU 102 by 

way of a power measurement iinlcl08 and an mtentipt Hnk 110. By way of the 
measurement Onl; 108. data indicating the power being consumed by the CPU 
102 is received by the local montoring drcuitiy 104. The Intenupt link 110 may 
be used to transmit inteinjpt messages to the CPU 102. Forexamplo.one 
intomqit message may be sent that activates an fntenupl handier to make a 
system cal to enable orTum on" power ihrotiflng mode at that CPU 102. 
Another imenvpt message may be sent that activates an htomipt handler to 
maice a qrstem caH to dbaWe or turn ofl" power throttling mode at Out CPU 
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102, Alternatively or In addltbn to the bitenupts, one or more specbl register 
106 mey be used to tum on and off the power-throttOno niode. The speetel 
reotsler 108 may be configured 80 88 to be readable by the CPU 102 and 
writable by the local drcuRry 104. Forexairple. eetdng a special reglstar 106 
5 >nay enable the powmrthrotlSngino(K>.whHecleartng the reQister 100 ma^ 
dteabia the power threitUng mode, fbe reglster<8) ld6 may be integrated wWi 
the CPU 102, or eltNnathrely» may be faceted external to the CPU 102. 

The local monllDffng dreuitiy 104 Is al$o coupled to the centtai 
power management modi4e (CPMM) 202. A management Rnk 114 between the 

10 CPMM 202 and the local monitoiingcirajlliy 104 may be used to receive a 
polling message liom the CPMM 202. When polled, the local manlloifng cfccuHiy 
104 may respond by transmlUIng via the management Unk 114 a mot mean 
squared or other derived power cansumptfcm value to the CPMM 20& fn one 
embodiment, the management lir* 114 may comprise an ouKof'band (OOB) Rnk, 

*5 such as an I2C (Inter IC) based bus. Preferably, the management IMc 114 is 
implemented to provide a relatively fast llnK that can be scaled either through the 
use of muKlplester or hube. other possible Imptemetitations of the management 
ft* 114 nny employ a serial bus. a USB (Unlvanal Serial Bus) connection, a 
1^ network, orothertype of Bnk. In addRlon.-lhe local drcullry 104 may notify 
20 the CPMM 202 ofanyfiaBureorotherwIse notable events fhfoi^h an mtemipt 
message. The fctloniipt message may be transmitted via a separate intarwpl 
Ine 116. or eltemat^eiy, may be seiK along the management Bnk 114. 

nG. 2 » a schematic diagram deplc^ a oentraRzad power 
management module (CPMM) 202 coupled to SSI dnwHiy lOO at muiO^e 
Z5 sen/era or computing de>^ces In eccordance vm en embodiment of the 
InveHtfon. As illustrated, there may be N servers, each with SSI circuitry 100, 
coupled to a Single CPMM 20Z The coupling between the CPMM 202 and each 
instance of SSI circuitry 100 may be implemented by way of e management Unk 
114 and en tntenupriine 119. 

30 The CPhttA 202 may be implemented using a management 

processor, a scalable management Ink to connect to the indivMual SSI drouitry 
100, and a IInk204toa oonlrol oonsoleoreooess to a local area neiMorfc(LAN) 

5 
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204. AnomeroomiTOAnieation link20« mivb© ti^ 

CPMM202 to other CPMMs 202 and/ortD power managomertsystems^ 
TDd CPMM 202 may be used to monllor the power beti^ 

conswnedbytheCPUs102. "mo monitoring may be don© t^ypoUlng. In 
i a^Wllton. local drwjjby 104 at the server or computing devk» may 

CPMM 202 of foihjre events qr other power related events by way of the Intenvpl 

linoii8. tftt» set of seivers or other computing devices being monltorod Is 

consuming power wHMn Ihe applicable limit, and no other power lelated 

condition exisis, than the CPMM 202 may eontkme polttng. 

On the other hand, if the eetof serasre or other computing devices 

b oonswnhg power boyond the applicable limit, or another power related 
condition exists, then the CPMM 202 may apply a oonf^uraUe set of mies to 
cause certain servers to operate In CPU thfotUiiq mode to mitigate the over- 
consumption or other power related condition. CPU thiotiUng mode refers to a 
mode in which a CPU te openatbig in a lower power consumption state with 
either lesser performance. Imlted functionality, or both. The request to enable 
the CPU throttling mode i^y be sent over the OOB management link 114. In 
aoooniance with one embodiment one or more computers may be set at a 

higher priortly level such that the CPUs 102 of other computers in the system 
have their power throltiedbefeie the higher prloraycompirterewSlmlte^ 
ordmerentsBtof mles maybeused to evaluate when and which servers or 
other computing devices to be teker* out of the CPU throttnng mode (and bade to 
the nonnal operating mode). In accordance with one embodlmant, there may be 
a oeitah level of hysteieelB between the oilea to enter and adt CPU thioitilng 
mode. l=or example, the power consumption level which induces thmttilng may 
be higher than the power consumption level at which thiottTng is removed. Such 
hysterssis would pi^svent Instances of rapid switching back and Ibith between 
CPU throtttag and nomwl modes in borderline circumstances. In one 
embodiment, Ihe CPMM 202 may be configured so that e user can interact with 
Ittoywayofaconsoteand8localornetwor1(edconsolellnk204. ThrDiflhsuch 
a local or remote console, a user may be enabled to setup or customize the 
afeiementiQned rules. oMah power consumption status or logs, and so on. 
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The CPMM 202 may Oso te oonfiffured to communicate wNh olher 
CPMMs 202. or power management systams. or other management syatems. by 
way Of another link 206. The other Hnk 206 may be Impiemented. for example, 
OS 8 LAN oonnodion. For example, each CPMM 202 may ooRect power 
consumption data or other infbmiation and forvna/d the fnfbrmation to a power 
management system or to a eystem admlnbtralor. The power management 
eystem may be conflgured for a user to view power consumption data, to modify 
orQvenfdethepowerthntaingnile8,and80on. Inthecasaoffaliureofasensr 
or coniputhg device, the asaocWed a»li*M aitt may receive 1^0^^ 
bttemjiit and may foniaid thenoHficatiDn upstream to a system admbHsbBtor (fbr 
example, byway a pagaror emaO). 

In accordance %vfHi en embodhnant of the Invention, Inaiead or all 
servere or computing devices on a rack indudbig the SSI drauitry IQO for power 
man^ment a subset of the servers or computing devices on a rack may 
IS include the SSI drcultiy 100 and be coupled to a CPMM 202. This enables 
power management of that subset of devices, particularly if lhat subset has a 
separate power supply system. 

In other embodiments, a dHfiarent grouping of systems, such as the 
computers In a duster or in a data center or tn an offlcQ. may each include SSI 
20 clniuliry 100 and be controlled by a CPMM 202. Such a grouping of systems 
may Indude, for instance, computen on more than one rack or blade diassis. or 
a gnxip of woricsiations in an office, in such embodiments, powerbad badandng 
maybe implemented, for example, to prevent hot spots of power utHizafion. 

In acconJance with another embodiment, the above design and 
25 architecture may |>e applied to a cooling (tastead of povMBr)JnftB5tfvictura. for 
example, hstead.of monitoring power consumpfion, the bcal monHoring drcuOiy 

l04maybeapplledtomQrtlortempefaiuiBsatthevartouscomputdis. Ifa 
temperature exceeds an aHowed level for a oomputar, the powMxthrottBng mode 
may be-acSvated for that computer ifi ^er Id oountersct the elevated 
so temperature. AltBinatrvely. or in addition. eddlUonal cooling via a fon or 

thennoolectrfc cooler may be eppUed m response to the devatsd temperatuie. 

In acconJanoe wfth another eml>odJment. each computer In a Qfoup 
may indtide more than one CPU. Power consumption at each ofihe CPUs may 
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be inonKored and a poweMhrottling inod» may be appHdd per CPU Hi each 
ooinputsr In other vwHds, the pneBenl bweniion is not Dintted to computers or 
aervers «vith one CPU each. 

In the above de&ctiption, numeraus specific details are given to 
S prevWe a thonsugh understanding of embodiments of the Invention. Hovyever. 
the above descrtptibn of iSustiated embbdimants of th6 Invention Is not Intended 
to be exhaustive or to lanit the InvenUon to the predse toms disclosed. One 
skfllad in the relevant art wi recognize that the Invention can be piacUoed 
without one or more of the specRIc details, or wHh other methods, oomponerjts, 
10 ete. In other Instant^ weNchownatiueturesoropeiHtions are not shown or 
desoribed in detaU to avoid obscuring aspects of the Invention. While spedflc 
embodiments of, and examples for, the Invention are described herein for 
illustrative purposes, varfous equivalent modificalions are posslble wilhin the 
scope of the invention, as those sUUed In the ralevant art will recognize. 
IS Th©» modifications can be made to the Invantton in l^ht of the 

above detailed description. The temis used in the fbliowing dalme should not be 
conslmed to limit the Bivention to the specific embodiments dlsdbsed In the 
specmcation and the daims. father, the soopa of the invention Is to be 
detennined by the foHowing dalms, %vhlch are to be oonstnjed in accordance 
20 with established doctrines of dalm Inteipietation. 



6 



PA(X13l16'RCVDAT2l28/20i)$ 2:58:36PM [EasteinStaiidardriinerSW 



FEB-28-2006 ■13:02" HP LEGfiL FT COLLINS 9708987247 



What is claimed is: 

5 

1. A8ystamfarpo)rarmanagainentofaorQupofeomputafs.iha8ystani 

Ooniprl&]n9[: 

( seiv^ side Inftasliuclura (SSI) diojltiy at each computer in th^ 

SSI Q'nwfiiy Indudihs local monilDiing drcuiliy coupled to a central 
w processing unit (CPU) of the computet: and 

a centralized power manasement module (CPMIM9 vvith a managamertt 

Ank to ttie SSI circuitry at each computer In the group, 
wheiBift the local circuHry at each computer in the group monitofB pmver 
consumption at the CPU and transmits power consunvAin data to 
« theCPMM.and 

««weln the CPMM appSes a set of rules to the power consumption data 

to detetmine when andat wMch oompiiers to enable and disable a 
CPU power thnntttlng mode. 

» 2. The ^yilsm of datml, wherein fte groups computers oornirises 
muittple servers mounlad in a rack. 

3. The system of darml.MhefeIn the group ofcompulem comprises a 
plurality of blade servsfs m a Uade chassis. 

as 

4. The system of claim l.whetem the group of oomputeraoompiftes a 
8<^ of workstations. 

5. The system of daim 1. further corrvrtsing: 

» a console coupled to the CPMM tor user imeractton. 

6. Trie intern of daim 5, wherein the console comprfses a oonsole 
connected bceliy to the CPMM. 
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7. "n^sy^en^ofdaimSrWheieinlhewnsoteainipilsesaiwiote 
coupled via a nefiwoilc to the CPMM. 

5 8. The system of daim 5, wherein tt» system Is configured to en*te a user 
to lha eforenwntiondd rules by wey of 9ie console. 

9. The system ofdalm 5, Mihemin the aysteffl is configured to enable a user 
to view power consumption dato by way of the console. 

10 

10. The system of claim 1, further comprtslng; 

addHJonal CPMMs wHh maneoement finks to SSI cUcutty at additional 

groups of computers; and 
a power management system coupled to the piutallty of CPMMs. 
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